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at least one natural language processing module to parse the first text to 
extract a keyword; 

a search engine to use the keyword as a search term and to return search 
results; 

at least one language translator to automatically translate the keyword into a 
plurality of languages prior to input to the search engine and to automatically 
translate the search results into second text in the identified language when the 
search results are in a plurality of languages; and 

at least one natural language generator to receive results to the command 
and to convert the second text into a natural language format according to the 
identified language. 

34. The system of claim 33, further comprising at least one automatic 
summarization module to automatically summarize the second text prior to 
converting the second text into the natural language format. 

35. The system of claim 33, further comprising at least one text to speech 
module to render the second text audibly to the user. 



REMARKS 

Reconsideration of the above referenced application in view of the following 
amendment and remarks is requested. Claims 1, 3-8, 13, 15-20, 25-28, 31, and 33- 
35 have been amended. Claims 2 and 14 have been cancelled. Existing claims 1, 
3-13, and 15-35 remain in the application. 
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ARGUMENT 

Claims 1-3, 5, 6, 9, 10, 12-15, 17, 18, 21, 22, 25, 26 and 28-30 are rejected 
under 35 USC 103(a) as being unpatentable over Junqua et al. (US 
6,324,51 2)(hereinafter Junqua) in view of Cohen et al (EP 1 014 277)(hereinafter 
Cohen). 

An embodiment of the present invention is a method and apparatus for a 
language independent, voice-based Internet or intranet search system. The present 
invention may be used to enrich the current Internet or intranet search framework by 
allowing users to search for desired information via their own native spoken 
languages. In one embodiment, the search system may accept voice input data 
from a user spoken in a conversational manner, automatically identify the language 
spoken by the user, recognize the speech in the voice input data, and conduct the 
desired search using the speech as input data for a search query to a search 
engine. To make the language independent voice-based search system even more 
powerful, several features may also be included in the system. Natural language 
processing (NLP) may be applied to extract the search terms from the naturally 
spoken query so that users do not have to speak the search terms exactly (thus 
supporting conversational speech). Machine translation may be utilized to translate 
search terms as well as search results across multiple languages so that the search 
space may be substantially expanded. Automatic summarization techniques may 
be used to summarize the search results if the results are not well organized or are 
not presented in a user-preferred way. Natural language generation and text to 
speech (TTS) techniques may be employed to present the search results back to 
the user orally in the user's native spoken language. The universal voice search 
concept of the present invention, once integrated with an Internet or intranet search 
engine, becomes a powerful tool for people speaking different languages to make 
use of information available on the Internet or an intranet in the most convenient 
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way. This system may promote increased Internet usage among non-English 
speaking people by making search engines or other web sites easier to use. 

In order to more particularly recite the present invention, the limitations of 
claim 2 have been incorporated into claim 1 . Claim 2 has been cancelled. Junqua 
is the primary reference relied on by the Examiner in rejecting the original claim 2. A 
thorough review of Junqua reveals that the cited reference teaches or suggests 
almost nothing about the invention as currently claimed. 

Junqua describes an interactive electronic program guide system using 
spoken language. Users can speak their commands to the audio/video system 
directly or through the telephone or the Internet. If the system does not have enough 
information necessary to perform the user's command, it prompts the user for 
additional information by first forming text queries and then sending text queries to a 
speech synthesizer, which converts these text queries into synthesized speech. The 
synthesized speech is played back to the user through a signal processing module. 
These text queries are in fact clarifications to the user's command. 

In Junqua's teaching, text queries are generated only when the system 
requires additional information from the user to complete the user command. They 
are totally different from the "results" in "receiving results to the command" as recited 
in claim 2, and they are not the same as the "second text" in claim 2 either. 

The "results" in "receiving results to the command" as recited in claim 2 are 
search results from the search engine, which are results of performing a completed 
user command. If the user prefers to listen to the search results, they are converted 
into a second text in* a natural language format, if necessary, through the automatic 
summarization module and the natural language generation module. Compared to 
the raw search results, this second text is more understandable when spoken. The 
second text is subsequently submitted to the text to speech module to be 
synthesized and further rendered to the user. 

In the Office Action, the examiner cites col. 3, lines 1-17 of Junqua as 
teaching the limitations of claim 2. This is erroneous; Junqua does not teach 
anything about the invention as claimed. Since Junqua does not teach or suggest 
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receiving results to the command and converting the results into a second text in a 
natural language format according to the identified language, Junqua does not teach 
or suggest amended claim 1 . No prima facie case of obviousness has been made in 
the Office Action dated July 8, 2002 for claim 1, as amended. Therefore, the 
amended claim 1 is allowable as presented. 

As to Claim 3, it depends from allowable claim 1 . Therefore, Claim 3 is also 
allowable. 

The Office Action relies on col. 3, lines 1-8 and col. 6, lines 30-46 of Junqua 
as teaching Claim 5. As mentioned above, the "queries" (col. 3. lines 1-8 of Junqua) 
are system prompts for additional information to complete a user command . The 
cited col. 6, lines 30-46 of Junqua teaches a natural language parsing process. The 
function of the natural language parser is to form a system understandable 
command based on the information in the user's spoken instructions. To achieve 
this goal, it uses a knowledge database and local and global parsers to extract key 
information to fill slots, or tags, in a frame structure of an acceptable command. 
However, it is not certain if the right tag information is extracted for each command. 
Several alternative candidates are identified for each tag, each assigned with a 
score or weight. The weight signifies how likely the tag information is correct. The 
overall weight for the entire frame structure of a command is calculated from weights 
of tags in the command frame. As a result, the output from the natural language 
parser is not a single command frame, but a list of ordered candidates. The 
candidate with the highest weight is on the top of the list. Usually only the top "N" 
candidates are selected to propogate through different processing stages. This is 
where the phrase "N-Best Candidates Selection" comes from. 

In marked contrast, in claim 5 of the present invention, the key word is used 
as a search query to a search engine. The results are the search results coming 

10 



09/685,419 

back from the search engine. In other words, the results are what the search engine 
has found. 

Junqua's system does not teach or suggest these limitations in claim 5. 
Therefore, claim 5 is allowable as presented. In addition, claim 5 depends from 
allowable claim 1 . 

Claim 6 is rejected based on Junqua, specifically col. 3, lines 18-34. The cited 
text of Junqua describes how the system interacts with the user via spoken 
language. When a user speaks his command to the system, the actual command 
may be embeded in complex sentences. The natural language parser extracts the 
key information from sentences to form a command understandable by the system. 
Based on what is found from a current copy of the electronic program guide, the 
dialog manager formulates a response in text format understandable by the user. 
This text response is further synthesized by the speech synthesizer and played back 
to the user. 

The limitations in claim 6, however, describe an automatic summerization 
module, which generates a short summary (gist) for the search results, if necessary. 
The user desired information may be scattered in very long search results. In this 
case, the automatic summarization module (36 in Fig. 1) produces a shorter 
summary of what has been found for the user. There are some other cases where 
automatic summarization proves very useful. For example, if the user prefers to 
listen to the search results while results include tables, graphs, or other unspeakable 
objects, the automatic summarization module, along with the natural language 
generation module, produces a summary text in a natural language format for 
contents in -these unspeakable objects. This summary text is then presented to the 
user in the synthesized speech format through the text to speech module. 

In the marked contrast, the summary (gist) provided by the automatic 
summarization module in the present invention is not the response, which tells the 
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user "that the program is, or is not, available for viewing on the requested day", in 
Junqua's system. These two different concepts are not even related. Throughout the 
entire patent, Junqua teaches or suggests nothing about automatic summarization. 
Therefore, claim 6 is allowable. 

Claim 9 is rejected based on Junqua. The examiner considers claim 9 to be 
shown inherently from Junqua because Junqua uses TCP/IP protocols for Internet 
access. In Junqua's teaching, the Internet is only used as a transmission means for 
remote access to the television set (col. 1, lines 37-44; col. 2, lines 28-38). Such a 
use of the Internet is very limited and fundamentally different from the use of the 
Internet through a web browser as described in the present invention. The major 
function of using a web browser is to provide a large knowledge reservior, not 
merely a transmission mechanism. A web browser is the interface to this reservior. 
Although Junqua teaches about TCP/IP, Junqualeaches or suggests nothing about 
a web browser. Since TCP/IP existed for many years before a web browser was 
invented, a reference to TCP/IP does not inherently teach or suggest a web 
browser. Therefore, claim 9 is not inherent from the cited text of Junqua. Hence, 
claim 9 is allowable. In addition to such a difference, claim 9 depends on claim 1, 
which is allowable. 

Claim 10 is rejected based on Junqua, specifically on col. 3, lines 1-16. This 
is erroneous. As mentioned above in the explanations regarding amended claim 1 , 
the "queries" in Junqua's teaching are given to the user by the system in an effort to 
complete the user's command by prompting the user for additional information. In 
contrast, in the present invention, the search query is a query (or key word(s)) 
submitted to the search engine. The search query may be directly from the user's 
instructions, or extracted from the user's instructions through a various number of 
processing steps, which may include language identification, speech recognition, 
natural language processing, and machine translation as shown in Fig. 1 in the 
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present invention. The web browser used in the present invention is an interface to 
the described search system which operates over the Internet. The "queries" taught 
in Junqua's system are totally different from the "query" described in the present 
invention. Junqua teaches or suggests nothing about the limitations in claim 10. 
Claim 1 0 is, therefore, allowable. 

The Office Action rejects claims 13-15 and 25-26 based on the same reasons 
used in the rejection of claims 1-3. 

Claims 13 and 25 are amended to incorporate the additional limitations in the 
amended claim 1. Claim 14 is cancelled and its limitations are incorporated into the 
amended claim 13. Claims 13 and 25 now have limitations similar to claim 1. 
Because amended claim 1 is allowable, claims 13 and 25 are allowable for the 
same explanations stated above for claim 1. Depending from claim 13, claim 15 is 
allowable. Claim 26 depends from allowable claim 25 and is also allowable. 

Claims 17-18, 21-22, and 28-30 are rejected for the same reasons used* in 
the rejection of claims 5-6 and 9-10. Based on the above explanations regarding 
claims 5-6 and 9-10, claims 17-18, 21-22, and 28-30 are now allowable. 

Claims 12 depends on claim 1 and claim 24 depends on claim 13. Both claim 
1 and claim 13 are allowable based on the above explanation regarding claim 1 and 
13. Claim 12 and 24 are thus also allowable. 

Claims 4, 7, 8, 16, 19, 20, 27, 31, and 33-35 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Junqua in view of Cohen, and further in view of 
Nosohara (EP 0 838 765) (hereinafter Nosohara). 

Claims 4 and 7 are amended to recite that the translation of the keyword is 
automatic. The addition of the limitation thus differentiates claims 4 and 7 from 
Nosohara's teaching. The Office Action cites col. 3, lines 6-22 of Nosohara as 
teaching claims 4, 7, and 8. After reading the entire document carefully, however, it 
is clear Nosohara requires that the destination language be specifically selected 
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before the translation means can translate a key word from the searcher's native 
language (col. 4, lines 32-57 of Nosohara, reads on "the language used to define 
the search conditions is selected by the searcher and then the databases to be 
searched (destination language) are designated ..."). In other words, the selection of 
the destination language for the key word translation in Nosohara is performed 
manually by the user. Nosohara teaches or suggests nothing about the limitations in 
the amended claims 4 and 7, which recite automatic destination language selection 
and translation. The amended claims 4 and 7 are, therefore, now allowable. 
Depending on allowable claim 7, claim 8 is also allowable. 

The Office Action rejects claims 16, 19, 20, 27, and 31 based on the same 
reasons used in the rejection of claims 4, 7, and 8. Based on the above 
explanations regarding claims 4, 7, and 8, claims 16, 19, 20, 27, and 31 are now 
allowable. 

As for claim 33, it is rejected based on the same reasons used in the 
rejection of claims 4, 7, and 8. Claim 33 recites a language independent voice 
based search system which comprises a language identifier, at least one speech 
recognizer, at least one natural language processing module, a search engine, at 
least one language translator, and at least one natural language generator. The 
limitations presented in amended claims 1, 4, 7, and 8, which are allowable, are 
incorporated into claim 33. None of the cited arts (Junqua, Cohen, and Nosohara) in 
the Office Action teach or suggest even a general search engine, let alone a 
complex language independent voice based search system with a number of 
limitations as recited in claim 33. No prima facie case of obviousness has been 
made in the Office Action dated July 8, 2002 for claim 33. Therefore, claim 33 is 
allowable as presented. 
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Claim 34 is rejected for the same reasons used in the rejection of claims 2 
and 6. Based on the above explanations regarding the claim 6 and original claim 2, 
claim 34 is also allowable. Claim 35, depending from claim 33, is also allowable. 

Claims 11, 23, and 32 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Junqua in view of Cohen, and further in view of well known prior 
art (MPEP 2144.03). 

Claim 11 depends on claim 9, claim 23 depends on claim 21, and claim 32 
depends on claim 29. Based on the same reasons stated above for claims 9, 21, 
and 29, claims 11, 23, and 32 are also allowable because they depend from 
allowable claims. 



In view of the foregoing, claims 1, 3-13, and 15-35 are all in condition for 
allowance. If the Examiner has any questions, the Examiner is invited to contact the 
undersigned at (503) 264-8074. Early issuance of Notice of Allowance is 
respectfully requested. 



CONCLUSION 
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Version with markings to show changes made: 

1 . (once amended) A method of interfacing to a system comprising: 
receiving voice input data from a user; 

identifying a language spoken by the user from the voice input data; 

converting the voice input data into a first text in the identified language by 
recognizing the user's speech in the voice input data based at least in part on the 
language identifier; 

parsing the first text to extract a keyword;[and] 

using the keyword as a command to an application; 

receiving results to the command: 

converting the results into a second text in a natural language format 
according to the identified language: and 

rendering the second text for perception by the user. 

2. Cancelled. 

3. (once amended) The method of claim [2]1_, wherein rendering comprises 
converting the second text into speech and rendering the speech to the user. 

4. (once amended) The method of claim 1, further comprising automatically 
translating the keyword into a plurality of languages other than the identified 
language and using the translated keywords as the command. 
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5. (once amended) The method of claim [2]1_, further comprising using the 
keyword as a search query to a search engine, wherein the results comprise search 
results from the search engine operating on the search query. 

6. (once amended) The method of claim [2]i, further comprising 
automatically summarizing the results prior to converting the results to the second 
text. 

7. (once amended) The method of claim [2]1_, further comprising 
automatically translating the keyword into a plurality of languages other than the 
identified language and using the translated keywords as a search query to a search 
engine, wherein the results comprise search results from the search engine 
operating on the search query. 

8. (once amended) The method of claim 7, further comprising automatically 
translating search results in languages other than the identified language into the 
identified language. 

9. The method of claim 1 , wherein the application comprises a web browser. 

10. The method of claim 9, wherein the web browser interfaces with a search 
engine and the command comprises a search query. 

11. The method of claim 9, wherein the web browser interfaces with a 
shopping web site and the command comprises at least one of a purchase order 
and a request for product information. 
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12. The method of claim 1, wherein the speech comprises conversational 
speech. 

13. (once amended) An article comprising: a storage medium having a 
plurality of machine readable instructions, wherein when the instructions are 
executed by a processor, the instructions provide for interfacing to a system by 
receiving voice input data from a user, identifying a language spoken by the user 
from the voice input data, converting the voice input data into a first text in the 
identified language by recognizing the user's speech in the voice input data based at 
least in part on the language identifier, parsing the first text to extract a keyword, 
[and] using the keyword as a command to an applicationM , receiving results to the 
command, converting the results into a second text in a natural language format 
according to the identified language, and rendering the second text for perception by 
the user. 



14. Cancelled. 

15. (once amended) The article of claim [14J13, wherein instructions for 
rendering comprise instructions for converting the second text into speech and 
rendering the speech to the user. 

16. (once amended) The article of claim 13, further comprising instructions 
for automatically translating the keyword into a plurality of languages other than the 
identified language and using the translated keywords as the command. 

17. (once amended) The article of claim [14]13, further comprising 
instructions for using the keyword as a search query to a search engine, wherein the 
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results comprise search results from the search engine operating on the search 
query. 

18. (once amended) The article of claim [14]13, further comprising 
instructions for automatically summarizing the results prior to converting the results 
to the second text. 

19. (once amended) The article of claim [14113, further comprising 
instructions for automatically translating the keyword into a plurality of languages 
other than the identified language and using the translated keywords as a search 
query to a search engine, wherein the results comprise search results from the 
search engine operating on the search query. 

20. (once amended) The article of claim 19, further comprising instructions 
for automatically translating search results in languages other than the identified 
language into the identified language. 

21 . The article of claim 13, wherein the application comprises a web browser. 

22. The article of claim 21, wherein the web browser interfaces with a search 
engine and the command comprises a search query. 

23. The article of claim 21, wherein the web browser interfaces with a 
shopping web site and the command comprises at least one of a purchase order 
and a request for product information. 
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24. The article of claim 13, wherein the speech comprises conversational 
speech. 

25. (once amended) A language independent voice based user interface 
system comprising: 

a language identifier to receive voice input data from a user and to identify 
the language spoken by the user; 

fal at least one speech recognizer to receive the voice input data and the 
language identifier and to convert the voice input data into first text based at least in 
part on the language identifier; 

fa] at least one natural language processing module to parse the first text to 
extract a keyword; and 

fa] at least one natural language generator to receive results to the command 
and to convert results into a second text [into]in a natural language format according! < 
to the identified language. 

26. (once amended) The system of claim 25, further comprising fal at least 
one text to speech module to render the second text audibly to the user. 

27. (once amended) The system of claim 25, further comprising fal at least 
one language translator to automatically translate the keyword into a plurality of 
languages, and to translate the second text into the identified language prior to 
converting the second text into the natural language format. 

28. (once amended) The system of claim 25, further comprising fanl at least 
one automatic summarization module to automatically summarize the second text 
prior to converting the second text into the natural language format. 
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29. The system of claim 25, wherein the system is coupled to a web browser. 

30. The system of claim 29, wherein the web browser interfaces with a 
search engine, the keyword comprises a search query for the search engine, and 
the second text comprises search results from the search engine. 

31 . (once amended) The system of claim 30, further comprising fal at least 
one language translator to automatically translate the keyword into a first plurality of 
languages for use as the search query, and to automatically translate the second 
text into the identified language prior to converting the second text into the natural 
language format, the second text comprising text in a second plurality of languages. 

32. The 1 system of claim 29, wherein the web browser interfaces with a 
shopping web site and the keyword comprises at least one of a purchase order and 
a request for product information. 

33. (once amended) A language independent voice based search system 
comprising: 

a language identifier to receive voice input data from a. user and to identify 
the language spoken by the user; 

fal at least one speech recognizer to receive the voice input data and the 
language identifier and to convert the voice input data into first text based at least in 
part on the language identifier; 

fa] at least one natural language processing module to parse the first text to 
extract a keyword; 
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a search engine to use the keyword as a search term and to return search 
results; 

[a] at least one language translator to automatically translate the keyword into 
a plurality of languages prior to input to the search engine and to automatically 
translate the search results into second text in the identified language when the 
search results are in a plurality of languages; and 

[al at least one natural language generator to receive results to the command 
and to convert the second text into a natural language format according to the 
identified language. 

34. (once amended) ThjTsWem of clajrfi 33, further comprising [anl at least 
one automatic summarization module to automatically summarize the second text 
prior to converting the second texihTtoYie natural language format. 

35. (once amended) The system of claim 33, further comprising fel at least 
one text to speech module to render the second text audibly to the user. 
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